Special Issue on “Digital Libraries in Medicine” Real-Time Digital Libraries Based on Widely Distributed, High Performance Management of Large-Data-Objects

We describe a distributed, wide area network based approach to collecting, cataloguing, storing, and providing Web access for large-data-objects that originate as high-speed data streams. Such data streams result from the operation of many types of on-line instruments and imaging systems, and are a “staple” of modern intelligence, scientific, and health care environments. The approach provides for real-time conversion of the data streams and large datasets to “objects” that are manageable, extensible, searchable, browsable, persistent, and available to both “ordinary” and high performance applications through the integration of a high-speed distributed cache and transparent management of tertiary storage. The user interfaces — for both application users and data collection curators — are provided by the capabilities of the World Wide Web. The capabilities of the architecture are not unlike a digital library system, and we give an example of a digital image library that has been built using this architecture. However, our approach particularly addresses the issues involved in creating such digital library-like collections automatically from the high date-rate, real-time output of, e.g., satellite imaging systems, scientific instruments, health care imaging systems, etc. We discuss the capabilities, architecture, and implementation of such a system, as well as several example applications in data-intensive environments. The applications include a metropolitan area ATM network based, on-line health care video imaging system, and several image database applications, including a photographic-image library (see [9]). We also describe the security architecture used to enforce data owner imposed access conditions.